DEDIS: Distributed Exact Deduplication for Primary Storage Infrastructures

Deduplication is now widely accepted as an efficient technique for reducing storage costs at the expense of some processing overhead, being increasingly sought in primary storage systems [7, 8] and cloud computing infrastructures holding Virtual Machine (VM) volumes [2, 1, 5]. Besides a large number of duplicates that can be found across static VM images [3], dynamic general purpose data from VM volumes allows space savings from 58% up to 80% if deduplicated in a clusterwide fashion [1, 4]. However, some of these volumes persist latency sensitive data which limits the overhead that can be incurred in I/O operations. Therefore, this problem must be addressed by a cluster-wide distributed deduplication system for such primary storage volumes. Although considerable space savings are obtainable, storage latency is critical in primary workloads so, deduplication must introduce negligible overhead to be viable. Traditional in-line deduplication includes computation inside the storage write path, adding unacceptable overhead in the latency of primary storage writes [6]. This penalty can be reduced by exploring data locality, however, it is only viable for specific storage workloads [5, 7]. On the other hand, off-line deduplication decouples aliasing from storage requests, reducing the latency penalty, but requiring additional temporary storage space and increasing the concurrency in storage accesses. In fact, given the overhead of copy-on-write mechanisms needed to avoid corrupting aliased data [1], even off-line deduplication must be confined to off-peak periods in order not to degrade latency. Off-peak periods in cloud infrastructures may be scarce, therefore